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PREFACE 


The  study  presented  in  this  Memorandum  is  part  of  the  RAND  Cost 
Analysis  Department's  continuing  development  of  cost  analysis  method¬ 
ology.  Specifically,  the  study  was  undertaken  to  examine  the  problems 
and  Issues  Involved  in  calculating  prediction  intervals  for  estimates 
that  are  suau  of  individually  derived  estimates  and  to  explore  the 
practical  implications  of  these  Issues  for  cost  analysis. 


SUMMARY 


This  Memorandum  sets  forth  methods  for  calculating  prediction 
intervals  for  total  estimates  that  are  sums  of  individually  derived 
estimates.  Special  attention  is  given  to  the  problem  encountered 
when  the  variances  of  each  of  the  individually  derived  estimates 
cannot  be  assumed  to  be  equal.  This  problem  is  essentially  Identical 
to  the  well-known  Behren-Fisher  problem  except  that  here  the  context 
is  one  of  deriving  a  "t-ratio"  for  summed  means  rather  than  for  the 
difference  between  means.  As  a  consequence,  the  prediction  interval 
for  the  case  with  unequal  variances  is  based  on  a  statistic  with  an 
approximate  t-distribution,  and  the  interval  itself  must  be  viewed 
as  an  approximation.  However,  for  purposes  of  practical  application, 
the  approximate  nature  of  these  intervals  should  cause  no  difficulty 
and  they  can  be  viewed  as  reasonably  accurate  representations  of  the 
true  intervals. 

Section  I  contains  the  introduction.  Section  II  describes  the 
standard  techniques  for  calculating  prediction  Intervals  for  indi¬ 
vidual  estimates,  and  Section  III  shows  the  development  of  the  formula 
for  calculating  the  prediction  interval  for  summed  estimates  when  the 
variances  of  the  individual  estimates  can  be  assumed  equal.  Section  IV 
addresses  the  problem  of  deriving  prediction  intervals  when  the  assump¬ 
tion  of  equal  variance  is  inappropriate.  Section  V  contains  examples 
and  discussion  for  each  of  the  cases  considered  in  the  preceding 


sections 
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I.  INTRODUCTION 


With  the  increased  application  of  system  and  program  analysis  to 
decision  problems,  the  development  and  use  of  statistically  derived 
cost  estimating  relationships  (CERs)  in  these  analyses  is  also  becoming 
more  common.  One  advantage  of  using  statistical  techniques  in  develop* 
lng  CERs  is  that,  in  the  process  of  deriving  the  relationships,  useful 
Information  about  the  reliability  of  the  CER  can  also  be  generated. 

The  coefficient  of  determination  (or  correlation  coefficient),  the 
standard  error,  the  "t"  statistics  for  each  of  the  coefficients  and 
other  similar  information  help  the  user  determine  how  much  confidence 
to  place  in  a  given  CER  as  an  estimating  tool. 

In  addition,  when  it  is  necessary  to  use  CERs  to  estimate  costs  for 
a  specific  system  with  given  characteristics,  it  is  also  possible  to 
make  probabilistic  statements  about  the  potential  magnitude  of  error 
associated  with  that  estimate.  Such  statements  are  usually  made  in 
the  form  of  prediction  Intervals,  delineated  by  lower  and  upper  bounds 
on  either  side  of  the  estimate,  with  an  average  probability  that  the 
actual  system  cost  will  lie  within  that  interval.  There  are,  of  course, 
many  sources  of  uncertainty  associated  with  such  estimates  that  are  not 
taken  into  consideration  in  prediction  Interval  calculations.  An 
example  of  one  is  the  uncertainty  associated  with  extrapolation,  and 
the  judgment  that  must  be  made  as  to  whether  the  relationship  is  valid 
outside  the  range  of  the  sample.  Nevertheless,  prediction  Interval 
statements,  when  properly  understood,  are  useful  tools  in  systems 
analysis. 
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In  practice,  although  what  is  ultimately  sought  are  estimates  of 
total  cost,  it  is  often  desirable  or  necessary  to  develop  the  CERs  for 
separate  subcomponents  or  subelements  which  are  then  summed.  This  need 
can  arise  for  various  reasons:  analytically,  it  is  useful  to  examine 
the  different  life  cycle  phases  separately,  and  therefore  to  develop 
separate  estimates  for  R6£>,  investment  and  operation  costs;  empirically, 
it  becomes  necessary  to  disaggregate  when  different  subcomponents  of 
cost,  such  as  R&D,  engineering,  and  production  materials,  relate  to 
different  variables  or  react  to  the  same  variables  differently. 


II.  PREDICTION  INTERVALS  FOR  SINGLE  ESTIMATES 


Assume  the  following  stray  of  observations  on  cost  and  a  corre¬ 


sponding  set  of  characteristics: 


yi  xil  xi2  *  *  *  Xip  (i=1>  •••*"> 


where  y^  =  the  ith  observation  on  the  cost  variable, 

■>  the  corresponding  ith  observation  on  each  of  the 
p  characteristics, 

n  *  number  of  observations, 

and  the  following  hypothesis 


yi  =  bo  +  bl  Xil  +  *  *  •  +  Vip  +  Zi 


where  »  observable  nonrandom  variables  (j*l,  .  .  •>  p) > 
b^  «  unknown  coefficients  (j“0 . p)  >  at*d 

«  unobservable  Independent  normal  random  variables 

with  zero  expected  value  and  a  constant  and 

2 

unknown  variance;  i.e.,  z^  ~  N(0,a  ). 

With  these  assumptions 


E(yilxi>  "  bo  +  bi  X11  +  *  •  *  +  Vip 


where  x^  -  the  vector  (x^,  x^ . xlp^’  and 

bj  -  the  unknown  coefficients  of  Eq.  (2). 
For  the  sake  of  brevity,  let 


ECyJx^  =  n£ 
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i  ! 
\ 

\  * 


bo  that  the  CER  derived  by  the  method  of  least  squares  may  be  denoted 

as 

*1  -  60  +  81  xij  +  •  •  •  +  £p  xip  (5) 

where  E( ^t)  *  by  hypothesis  and 

bj  -  the  estimated  parameters  of  Eq.  (3)  chosen  so 

n  2 

as  to  minimize  E  (y.  -  (1  )  . 

1-1  1  1 

From  the  hypothesis  and  the  use  of  the  method  of  least  squares  it 
follows  that 

,  2 


ra  ~  X 


(6) 


l.e.,  Eq.  (6)  has  a  chi-square  distribution  with  r  degrees  of  freedom, 

where  r  -  n-(p+l)  and 

-  2 

,  n  (y  -  u.) 

o  -  £  — - i — 


i-1 

It  also  follows  that 


~  N(0,1) 


(7) 


ao 


A  2  2  A  ★★ 

since  E(y^  -  n^)  -  0  and  a  a  -  varty^  - 


Kendall,  M.  G.  and  A,  Stuart,  The  Advanced  Theory  of  Statistics, 
Fol.  2,  Harper  Publishing  Co.,  New  York,  p.  83. 

Ibid.,  p.  363,  where  a^  -  [ 1  +  i  +  x'A'^x],  A  equals  (n-l)  times 

the  covariance  matrix  of  the  variables  x  .  (i-1,  .  .  .  n)  and  (j-1,  .  . 

VI- 


p) . 


and  x’  equals  [(x^  -  x^),  .  .  .  (x^ 
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Divlding  Eq.  (6)  by  its  degrees  of  freedom,  taking  the  square  root 
of  this  variable  and  dividing  it  into  Eq.  (7)  results  in  a  variable  with 
a  t  distribution  with  r  degrees  of  freedom: 


aa 


(8) 


The  probability  statement  for  the  prediction  interval  based  on 
this  t  statistic  is 


P<-*W2  <  tr<t*/2>  "  1  '  °  (9) 

where  a  is  the  level  of  significance  or  average  probability  that  t^ 
will  lie  outside  the  interval  [ ~ta/2 ■  12^ ' 

Rewriting  Eq.  (9)  by  substituting  Eq.  (8)  gives 

P(-'«/2  *  ^*4^.  '  W  ‘  1  -  “  (1°> 

oa 


so  that 

P(ii  -  5atcr/2  <  yi  <  \  +  5ato/2)  “  1  "  *  <ll> 

and  the  prediction  interval  for  at  the  a  level  of  significance  is 

da 


“i  *  ta/2 


(12) 
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III.  SUMMED  TOTALS  WITH  EQUAL  VARIANCES 


Now  we  will  consider  Che  case  where  a  prediction  interval  Is  desired 
for  a  total  cost  estimate  which  is  the  sum  of  m  subelements  and  the 
variances  for  each  of  these  m  elements  are  equal.  Let 


M. 


i-1 


ik 


(13) 


and 


m 

£ 

i-1 


rik 


(14) 


where  the  subscript  k  (k-1,  .  .  . ,  m)  has  been  added  to  indicate  that 
the  and  y^  are  values  for  the  kth  subelement  of  the  total  and  that 
there  are  now  m  sets  of  hypotheses  such  as  that  for  Eq.  (2)  and  m 
conditional  means  estimated  from  m  distinct  CERs,  such  as  that  shown 
In  Eq.  (5). 

Assuming  that  the  distributions  of  the  conditional  means,  > 

are  Independent,  the  following  is  a  normally  distributed  random 

* 

variable  with  zero  mean  and  unit  variance. 


The  assumption  that  the  distributions  are  independent  will  be 
maintained  throughout  the  development  of  prediction  interval  formulas. 
While  there  is  evidence  that  in  practice  there  will  be  a  tendency  toward 
negative  correlation  in  the  types  of  problems  for  which  these  prediction 
Interval  formulas  are  likely  to  be  used,  this  will  be  treated  as  essen¬ 
tially  an  ad  hoc  problem  in  this  Memorandum.  Since  the  observed  sample 
residuals  for  each  of  the  m  subelement  CERs  are  hypothesized  to  be  drawn 
from  a  normally  distributed  population,  it  is  possible  to  test  the  as¬ 
sumption  of  independence  by  testing  for  correlation  between  the  sets 
of  residuals  (see  p.  14).  Thus,  it  is  possible  to  assess  the  applica¬ 
bility  of  the  assusption  in  each  case.  Further,  in  Instances  where 
the  negative  correlation  is  pronounced,  its  effect  should  be  adverse 
enough  on  the  results  of  the  regression  analysis  to  discourage  the  use 
of  the  related  CERs. 


-7- 


Y1  -Mi 
2  2 


Adding  the  subscript  "k"  to  Eq.  (6)  to  Indicate  that  It  corresponds 
to  the  k1*1  component  of  the  summed  total  gives 


rk  °k  2 

^  X  • 


Summing  the  expression  In  Eq.  (16)  over  the  m  subelements  gives 


5  l\< 


k-ll  2 


Since  by  assumption  the  variances  are  equal,  Eq.  (17)  becomes 


i  m  n 

—  *  ('k  ”k> 
a  k«l 


which  because  of  the  assumption  of  Independence  has  a  chi-square 
distribution  with  r (■£  r.  )  degrees  of  freedom.  Similarly,  Eq.  (15) 

le  K 


becomes 


Y1  -M1 


°V7*r 


Dividing  Eq.  (18)  by  Its  degrees  of  freedom,  taking  the  square 
root  and  dividing  Into  Eq.  (19)  gives 
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tr  .  <Yl  -  V  <»>*  (20) 

<*k 

which  has  a  "t"  distribution  with  r  degrees  of  freedom. 

The  corresponding  prediction  interval  is: 


(21) 


i 

I  1 


IV.  SUMMED  TOTAL  WITH  UNEQUAL  VARIANCES 


If  the  unknown  variances  In  the  denominator  of  Eq.  (17)  are  not 
equal,  then  the  sum  Is  a  linear  combination  of  chi-squares  rather  than 
the  sum  of  multiples  of  a  chi-square,  and  therefore,  no  longer  a  chi- 
square.  Consequently,  even  though  Eq.  (15)  still  has  a  normal  distri¬ 
bution  with  zero  mean  and  unit  variance,  the  ratio  Bhown  in  Eq.  (20) 
will  no  longer  have  a  "t"  distribution  and  standard  techniques  for 
deriving  prediction  intervals  are  no  longer  valid.  It  is  possible, 
however,  to  develop  approximate  prediction  interval  formulas  based  on  a 
statistic  which  has  the  approximate  distribution  required.  The  basic 
procedure  is  to  define  a  variable  which  has  the  same  first  moment  as 
the  chi-square  divided  by  its  degrees  of  freedom  and  to  estimate  the 
degrees  of  freedom  for  which  this  variable  has  approximately  a  chi-square 
distribution  by  matching  the  second  moments.*  This  variable  then  can  be 
used  in  developing  a  variable  which,  in  turn,  has  approximately  a  "t" 
distribution. 

Since  the  variable  in  Eq,  (15)  has  a  normal  distribution  with 
zero  mean  and  unit  variance,  it  follows  that 


This  technique  was  worked  out  for  the  two-variance  case  by 
B,  L.  Welch  ("The  Significance  of  the  Difference  Between  Two  Means 
When  the  Population  Variances  are  Unequal,"  Biometrlka.  No.  29,  1938, 
p.  350)  and  is  described  by  M,  G.  Kendall  and  A.  Stuart  in  The  Advanced 
^•0^[^«tistica.  Vol.  2,  Harper  Publishing  Co.,  New  Yo^T,'  Wii, - 
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Define  a  new  variable 


0^  and 


Its  estimated  value 


-2 

6 


as  follows: 


and 


then 


o  2 

6'  "  ?  \ 
k  * 


2 

“k 


*2 

- 


r  A2 

k  k 


2 


2  9  a 

Since  E(ok)  ■  a*,  It  follows  that  E(9)  -  0; 
so  that 


1  . 


Matching  the  second  moments: 


(?3) 


(24) 


(25) 


(26) 


(27) 


and  solving  for  the  degrees  of  freedom  gives 


Z 

k 


9 


\ 


■*&*&*' 


x  m 


(28) 
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V.  EXAMPLE 


This  section  contains  an  example  of  the  use  of  formulas  presented 
In  the  preceding  sections,  highlighting  the  significant  features  of 
the  prediction  interval  approximation  described  in  Section  IV.  Modi¬ 
fied  data  on  direct  base  maintenance  personnel,  taken  from  RM-4748-PR, 

* 

are  used  In  the  example.  The  RM  presents  CERs  for  Organizational 
Maintenance  Manhours  (ODM) ,  Field  Maintenance  Manhours  (FDM),  the  sum 
of  ODM  and  FDM  (0  4-  FDM) ,  Communications-Armament-Electronics  Mainte¬ 
nance  Manhours  (CAE)  and  Total  Maintenance  Manhours  (TDM),  the  sum  of 
ODM,  FDM,  and  CAE. 

For  the  example,  these  same  CERs  were  recalculated,  dropping  the 
one  classified  data  point.  The  data  required  for  the  calculation  of 
the  prediction  intervals,  based  on  these  results,  are  shown  in  Table  1. 
Lines  1,  2,  and  4  represent  subcomponents  which  would  normally  be  added 
together  in  an  analysis  of  total  costs  and  comprise  the  type  of  summed 
total  of  central  interest  in  this  Memorandum.  However,  CERs  for  the 
subtotal  of  ODM  and  FDM  as  well  as  the  total  sum  of  ODM,  FDM,  and  CAE 
were  also  calculated  in  the  referenced  Memorandum,  as  indicated  on 
lines  3  and  5,  respectively,  of  Table  1.  As  a  consequence,  it  is 
possible  not  only  to  compare  the  difference  between  the  results  for 
the  summed  totals,  with  and  without  the  assumption  of  constant  variance, 
but  also  to  compare  the  difference  between  intervals  for  the  su toned 
individual  estimates  and  the  direct  estimate  of  the  summed  total. 

* 

Dlenemann,  P.  F. ,  and  G.  C.  Sumner,  Estimating  Aircraft  Base 
Maintenance  Personnel.  RM-4748-PR,  The  RAND  Corporation,  October  1965. 
The  data  used  are  on  maintenance  manhours  per  flying  hour  for  fighter 
aircraft  as  shown  on  pp.  24-33. 


'  fc-C.v  • 
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Table  1 

DATA  FOR  PREDICTION  INTERVAL  CALCULATION 


Sample 

Mean 

Coefficient  of 
Determination 

Standard 

Error 

Number  of 
Observations 

Item 

<\> 

(R2) 

{nk> 

ODM 

13.556 

.923 

.922 

9 

FDM 

12.333 

.958 

.735 

9 

(0+F)DM 

25.889 

.986 

.890 

9 

CAE 

7.111 

.808 

2.345 

9 

Degrees  of 
Freedom 


<rk> 


6 

6 

5 

7 


33.000 


962 


2.488 


9 


5 
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To  test  the  assumption  of  the  Independence  of  the  distributions 
of  the  conditional  means,  simple  regressions  were  run  on  the  observed 
residuals  of  each  of  the  pairwise  combinations:  the  residuals  of  the 
ODM  regression  were  run  against  the  residuals  of  the  FDM  regression, 
etc.  The  resulting  Bimple  correlations  are  tabulated  below: 


Residuals  on 
Variables 

ODM  and  FDM 

ODM  and  CAE 

FDM  and  CAE 


Simple 

Correlations 

-.043 

.149 

.321 


Since  the  residual  values  are  assumed  to  be  normally  distributed,  the 

hypothesis  that  distributions  are  independent  can  be  tested  with  the 

hypothesis  that  the  correlations  are  zero.  The  critical  value  for 

this  test  at  the  five-percent  level  of  significance  is  . 798  for  7  de- 
* 

grees  of  freedom.  None  of  the  sample  correlations  are  greater  than 
this  value,  so  the  hypothesis  of  independence  cannot  be  rejected. 

In  deciding  whether  or  not  the  assumption  of  equal  variance  is 
appropriate  one  may  either:  (1)  test  for  equality  of  variance  and  use 
the  approximation  only  if  the  hypothesis  of  equal  variance  must  be 
rejected:  or  (2)  make  no  attempt  to  bring  in  the  assumption  of  equality, 
and  use  approximation  of  Section  III  regardless  of  the  values  of  the 
sample  variances.  As  the  example  will  illustrate,  the  difference  in 
the  results  will  be  slight  when  the  sample  variances  are  nearly  equal, 

★ 

Dixon,  W.  J.,  and  F.  J.  Massey,  Introduction  to  Statistical 
Analysis ,  McGraw-Hill  Book  Company,  Inc.,  New  York,  1959,  Table  30-a, 
p.  468. 
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regardless  of  which  technique  for  calculating  the  intervals  is  used. 
However,  because  of  the  basic  weakness  of  variance  tests  with  small 
samples,  the  first  approach  is  not  recommended  for  small  sample  cases. 

For  purposes  of  illustration,  however,  tests  will  be  performed 
on  the  variances  in  our  example  regardless  of  the  small  sample  size 
and  prediction  interval  results  using  both  techniques  will  be  presented. 
Considering  the  case  for  the  sum  of  ODM  and  FDM  first,  the  null 
hypothesis  is  var(ODM)  »  var(FDM)  and  the  test  statistic  is: 

-2 

aODM  =  .850  ,, 

«2  .540  *  * 

CTFDM 

The  critical  F  value  at  the  5-percent  level  of  significance  is  4.39. 
Thus,  the  null  hypothesis  is  not  rejected. 

Consider  next  the  sum  of  ODM,  FDM,  and  CAE.  In  a  case  with  more 
than  two  subelements ,  the  hypothesis  of  equal  variance  may  be  tested 
by  testing  for  equality  between  the  largest  sample  variance  and  the 
smallest.  In  our  example,  the  null  hypothesis  for  the  assumption 
that  var(ODM)  *  var(FDM)  **  var(CAE)  is  simply  var(CAE)  «  var(FDM)  and 
the  test  statistic  is: 


q2CAE 

52FDM 


5.499 

ow 


10.2 


so  that  the  null  hypothesis  is  rejected. 

On  the  basis  of  these  tests,  it  appears  that  the  assumption  of 
equality  of  variance  is  warranted  for  the  sum  of  ODM  and  FDM  but  not 
for  the  sum  of  ODM,  FDM,  and  CAE.  However,  Table  2  shows  the  calculated 


Table 


prediction  Intervals  for  each  sum  under  both  assumptions.  To  facilitate 
comparisons,  all  "predicted"  values  were  calculated  using  the  sample 
mean  value  of  each  of  the  independent  variables.  Because  of  the 
properties  of  the  least-squares  estimating  relationships,  the  pre¬ 
dicted  values  are,  therefore,  equal  to  the  corresponding  mean  value 
of  each  of  the  dependent  variables.  In  addition  to  calculating  the 
five  cases  for  which  CERs  exist,  prediction  Intervals  were  calculated 
for  three  additional  combinations.  These  are  shown  in  lines  6,  7,  and 
8  of  Table  2. 

Perhaps  the  most  dominant  feature  of  the  intervals  for  the  summed 
totals  in  Table  2  is  that  the  approximation  is  always  somewhat  more 
conservative  but  that  the  results  are  not  strikingly  different.  How¬ 
ever,  several  additional  general  observations  can  be  made.  When  the 
assumption  of  equality  appears  warranted  because  of  the  near  equality 
of  the  sample  variances,  the  interval  for  the  summed  total  is,  approx¬ 
imately,  ta/2  tim€S  the  square  root  of  the  number  of  subelements 
squared  times  their  weighted  average  variance.  When  the  assumption 
of  equality  is  not  warranted,  the  worst  case  for  the  approximation 
occurs  if  there  is  significant  disparity  in  both  the  sample  variances 
and  the  number  of  degrees  of  freedom  in  each  sample. 

As  hinted  in  the  results  of  Table  2,  the  number  of  degrees  of 
freedom  in  the  smallest  sample  sets  the  lower  limit  for  the  degrees 
of  freedom  for  the  approximation.  The  limits  of  the  degrees  of  freedom 
for  the  approximation  formula  may  be  expressed  as  min  rfc  «  r  «  £  rk« 

As  the  disparity  in  the  sample  variances  Increases  the  lower  limit  is 
approached.  Thus,  in  line  3,  where  the  difference  in  the  sample  vari¬ 
ances  is  about  57  percent,  there  is  a  reduction  of  degrees  of  freedom 
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of  25  percent,  from  12  to  9.  In  line  6,  however,  the  difference  is 
over  600  percent  and  the  reduction  in  the  degrees  of  freedom  is  from 
13  to  the  lower  limit,  6,  which  in  this  case  is  about  a  50  percent 
reduction. 

In  comparing  the  results  of  the  intervals  for  the  summed  total 
with  the  intervals  for  the  direct  estimates,  it  can  be  seen  that  the 
summed  total  intervals  may  be  either  larger  (line  3)  or  smaller  (line 
5).  It  is  difficult  to  establish  definitive  generalizations  as  to 
which  total  will  have  the  smallest  prediction  interval,  but  one  inter¬ 
esting  fact  is  that  the  existence  of  Intervals  of  unacceptable  size 
on  a  subelement  will  not  necessarily  produce  unacceptable  results  for 
the  summed  total.  The  intervals  for  CAE  (line  3)  are  almost  as  large 
as  the  estimate  for  CAE.  However,  the  intervals  for  the  summed  total 
(line  5)  are  only  about  20  percent  on  either  side  of  the  estimate. 

This  difference  is  due  to  the  fact  that  CAE  is  a  small  percentage  of 
the  total  and  that  the  intervals  (and  sample  variances)  for  the  other 
elements  are  relatively  small.  This  demonstrates  that  it  is  possible 
to  develop  statistically  acceptable  estimates  of  total  cost,  as 
measured  by  the  prediction  intervals,  by  subcomponent,  even  when  it 
is  not  possible  to  derive  strong  estimating  relationships  for  every 
subelement  in  the  total.  Thus,  it  is  not  always  necessary  or  desirable 
to  discard  useful  Information  about  subelement  costs  by  aggregating  to 
higher  levels  when  difficulty  is  encountered  in  individual  sublement 
CERs. 
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